BENCHIP: Benchmarking Intelligence Processors

摘要

The increasing attention on deep learning has tremendously spurred the designof intelligence processing hardware. The variety of emerging intelligenceprocessors requires standard benchmarks for fair comparison and systemoptimization (in both software and hardware). However, existing benchmarks areunsuitable for benchmarking intelligence processors due to their non-diversityand nonrepresentativeness. Also, the lack of a standard benchmarkingmethodology further exacerbates this problem. In this paper, we proposeBENCHIP, a benchmark suite and benchmarking methodology for intelligenceprocessors. The benchmark suite in BENCHIP consists of two sets of benchmarks:microbenchmarks and macrobenchmarks. The microbenchmarks consist ofsingle-layer networks. They are mainly designed for bottleneck analysis andsystem optimization. The macrobenchmarks contain state-of-the-art industrialnetworks, so as to offer a realistic comparison of different platforms. We alsopropose a standard benchmarking methodology built upon an industrial softwarestack and evaluation metrics that comprehensively reflect the variouscharacteristics of the evaluated intelligence processors. BENCHIP is utilizedfor evaluating various hardware platforms, including CPUs, GPUs, andaccelerators. BENCHIP will be open-sourced soon.